Overlay networks are used to enable network virtualization built on top of existing physical networks. An overlay network is a network virtualization technology that achieves multi-tenancy in a computing environment. An overlay network stretches a Layer 2 (L2) network across Layer 3 (L3) boundaries of the physical infrastructure by decoupling the physical topology from the logical topology. The decoupling provides flexibility and mobility allowing compute nodes to be dynamically placed anywhere in a data center, removing the traditional L2 boundaries of the physical infrastructure.
Examples of overlay networks include Virtual eXtensible LAN (VXLAN), Generic Network Virtualization Encapsulation (GENEVE), and Network Virtualization using Generic Routing Encapsulation (NVGRE). For instance, VXLAN is an L2 overlay scheme over L3 network. VXLAN encapsulates an Ethernet L2 frame in IP (MAC-in-UDP encapsulation) and allows VMs to be a part of virtualized L2 subnets operating in separate physical L3 networks. Similarly, NVGRE uses Generic Routing Encapsulation (GRE) to tunnel L2 packets over L3 networks.
Many packet processing frameworks utilize processors with multiple cores and operate on the assumption that a flow (identified by the 5-tuple of IP source, IP destination, Layer 4 (L4) source, L4 destination, and protocol identification) will arrive on the same processing core to avoid the cost of locking and synchronization that is needed to correctly process the packets of the same flow. To dispatch a flow, hardware receive side scaling (RSS) is used. The RSS hashes a packet based on the 5-tuple and then uses the hash to select a core for receive processing to provide maximum parallelism. All modern network interface cards (NICs) can perform RSS by identifying the 5-tuple from a packet that does not have overlay network encapsulation and do proper hashing.
The overlay mechanism, however, adds additional encapsulation layers over a packet. Many NICs do not consider the overlay network encapsulation when hashing the packets received from an overlay network. The packets of a flow that are received from the overlay network and have overlay network encapsulation may therefore land on a different core than the packets of the same flow received from a portion of the network such as VLAN that does not perform overlay encapsulation.